{
    "componentChunkName": "component---src-templates-post-tsx",
    "path": "/Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition/",
    "result": {"data":{"logo":null,"markdownRemark":{"html":"<h1>Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition</h1>\n<p>Yu Zhang et al., 2020<br>\nGoogle Research, Brain Team</p>\n<hr>\n<h2>Reference</h2>\n<ul>\n<li><a href=\"https://github.com/kakaobrain/nlp-paper-reading/blob/master/notes/wav2vec%202.0.md\">Wav2vec 2.0 Summary</a></li>\n<li><a href=\"https://github.com/speech-paper-reading/speech-paper-reading/blob/main/notes/conformer.md\">Conformer Summary</a></li>\n</ul>\n<hr>\n<h2>Summary</h2>\n<ul>\n<li>현재 Papers with Code 기준 ASR 부문 State-Of-The-Art</li>\n<li>Conformer + Wav2vec 2.0 + Noisy Student Training</li>\n<li>1.3%/2.6%/1.4%/2.6% on the dev/dev-other/test/test-other sets</li>\n</ul>\n<img src=\"https://user-images.githubusercontent.com/42150335/111322357-2f3dac80-86ac-11eb-8a05-24be848077de.png\" height=\"300\">\n<hr>\n<h3>Wav2vec 2.0 Pre-training</h3>\n<img src=\"https://user-images.githubusercontent.com/42150335/92450554-8a22b280-f1f6-11ea-8f66-0616b29d8c94.png\">\n<ul>\n<li>53,000이라는 대량의 Unlabeled speech data로 학습</li>\n<li>Pre-training 과정\n<ul>\n<li>Waveform에서 CNN을 이용해서 피쳐를 뽑음</li>\n<li>이를 Vector Quantization을 통해 one-hot-vector로 만들고 Embedding matrix를 내적하여 token화 함</li>\n<li>일정 비율로 Masking하고 다음 Token이 뭔지 알아맞추게하는 Masked Language Modeling (MLM) 학습 방식 적용</li>\n</ul>\n</li>\n<li>Vector Quantization</li>\n</ul>\n<img src=\"https://camo.githubusercontent.com/4e4253817961b5bead8072739c39bd3f3daaced98e8735018c50e8a55d78fb9c/68747470733a2f2f692e696d6775722e636f6d2f7931355175355a2e706e67\" height=\"300\">\n  - Z를 선형변환하여 logit을 만듦\n  - 여기에 Gumbel Softmax와 argmax를 취해 one-hot vector를 만듦\n  - 이후 Embedding matrix를 내적해 Z^를 만듦\n<h3>Conformer</h3>\n<ul>\n<li>Self-attention 기반한 트랜스포머는 global-context 정보를 잘 표현하지만, local-context에서는 부족하다는 단점이 있음</li>\n<li>반면, CNN 기반 모델은 local-context는 잘 표현하지만 global-context를 반영하기 위해서는 적당한 dilation과 깊은 구조를 가져야 함</li>\n<li>이 두 방법을 결합하여 global-context와 local-context 모두 잘 표현할 수 있도록 하기 위한 transformer + CNN 결합구조인 Conformer 구조 제안</li>\n</ul>\n<h2>Method</h2>\n<p>본 논문에서 실험한 모델 구조 및 트레이닝 방법</p>\n<h3>Model Architecture</h3>\n<p>Conformer Encoder + LSTM decoder로 이루어진 Transducer 구조</p>\n<img src=\"https://user-images.githubusercontent.com/42150335/111322627-77f56580-86ac-11eb-957c-d51db823e4e4.png\" height=\"600\">\n<ul>\n<li>기존 Conformer 구조에서 Wav2vec 2.0 Pre-training을 도입하기 위해 Masking하는 과정과 Linear Layer (Quantization 대체) 추가</li>\n<li>사이즈별로 L, XL, XXL로 구분</li>\n<li>XXL+는 Conformer XXL에 Conformer block을 stack (XXL보다 50M 파라미터 추가)</li>\n</ul>\n<img src=\"https://user-images.githubusercontent.com/42150335/111324910-8ba1cb80-86ae-11eb-997f-12634e7b6164.png\">\n<h3>Wav2vec 2.0 Pre-training</h3>\n<ul>\n<li>60k Libri-Light 데이터셋 사용</li>\n<li>기존 논문과 달리, 인풋으로 log-mel spectrogram 사용</li>\n<li>Masking 된 인풋과 예측한 context vector 간의 contrastive loss로 학습</li>\n</ul>\n<h3>Noisy Student Training with SpecAugment</h3>\n<img src=\"https://user-images.githubusercontent.com/42150335/111328868-fa345880-86b1-11eb-925c-a76cdbfd7c8c.png\" height=\"200\">\n<h2>Experiiments</h2>\n<img src=\"https://user-images.githubusercontent.com/42150335/111329076-2e0f7e00-86b2-11eb-8c87-17d2eca8948b.png\" height=\"400\">\n<ul>\n<li>결과적으로 Pre-training + NST가 좋은 성적을 냄</li>\n</ul>","htmlAst":{"type":"root","children":[{"type":"element","tagName":"h1","properties":{},"children":[{"type":"text","value":"Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Yu Zhang et al., 2020"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\nGoogle Research, Brain Team"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"hr","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Reference"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://github.com/kakaobrain/nlp-paper-reading/blob/master/notes/wav2vec%202.0.md"},"children":[{"type":"text","value":"Wav2vec 2.0 Summary"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://github.com/speech-paper-reading/speech-paper-reading/blob/main/notes/conformer.md"},"children":[{"type":"text","value":"Conformer Summary"}]}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"hr","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Summary"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"현재 Papers with Code 기준 ASR 부문 State-Of-The-Art"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Conformer + Wav2vec 2.0 + Noisy Student Training"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"1.3%/2.6%/1.4%/2.6% on the dev/dev-other/test/test-other sets"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/111322357-2f3dac80-86ac-11eb-8a05-24be848077de.png","height":300},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"hr","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Wav2vec 2.0 Pre-training"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/92450554-8a22b280-f1f6-11ea-8f66-0616b29d8c94.png"},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"53,000이라는 대량의 Unlabeled speech data로 학습"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Pre-training 과정\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Waveform에서 CNN을 이용해서 피쳐를 뽑음"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"이를 Vector Quantization을 통해 one-hot-vector로 만들고 Embedding matrix를 내적하여 token화 함"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"일정 비율로 Masking하고 다음 Token이 뭔지 알아맞추게하는 Masked Language Modeling (MLM) 학습 방식 적용"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Vector Quantization"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://camo.githubusercontent.com/4e4253817961b5bead8072739c39bd3f3daaced98e8735018c50e8a55d78fb9c/68747470733a2f2f692e696d6775722e636f6d2f7931355175355a2e706e67","height":300},"children":[]},{"type":"text","value":"\n  - Z를 선형변환하여 logit을 만듦\n  - 여기에 Gumbel Softmax와 argmax를 취해 one-hot vector를 만듦\n  - 이후 Embedding matrix를 내적해 Z^를 만듦\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Conformer"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Self-attention 기반한 트랜스포머는 global-context 정보를 잘 표현하지만, local-context에서는 부족하다는 단점이 있음"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"반면, CNN 기반 모델은 local-context는 잘 표현하지만 global-context를 반영하기 위해서는 적당한 dilation과 깊은 구조를 가져야 함"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"이 두 방법을 결합하여 global-context와 local-context 모두 잘 표현할 수 있도록 하기 위한 transformer + CNN 결합구조인 Conformer 구조 제안"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Method"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"본 논문에서 실험한 모델 구조 및 트레이닝 방법"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Model Architecture"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Conformer Encoder + LSTM decoder로 이루어진 Transducer 구조"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/111322627-77f56580-86ac-11eb-957c-d51db823e4e4.png","height":600},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"기존 Conformer 구조에서 Wav2vec 2.0 Pre-training을 도입하기 위해 Masking하는 과정과 Linear Layer (Quantization 대체) 추가"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"사이즈별로 L, XL, XXL로 구분"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"XXL+는 Conformer XXL에 Conformer block을 stack (XXL보다 50M 파라미터 추가)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/111324910-8ba1cb80-86ae-11eb-997f-12634e7b6164.png"},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Wav2vec 2.0 Pre-training"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"60k Libri-Light 데이터셋 사용"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"기존 논문과 달리, 인풋으로 log-mel spectrogram 사용"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Masking 된 인풋과 예측한 context vector 간의 contrastive loss로 학습"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Noisy Student Training with SpecAugment"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/111328868-fa345880-86b1-11eb-925c-a76cdbfd7c8c.png","height":200},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Experiiments"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/111329076-2e0f7e00-86b2-11eb-8c87-17d2eca8948b.png","height":400},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"결과적으로 Pre-training + NST가 좋은 성적을 냄"}]},{"type":"text","value":"\n"}]}],"data":{"quirksMode":false}},"excerpt":"Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition Yu Zhang et al., 2020 Google Research, Brain Team Reference…","fields":{"readingTime":{"text":"3 min read"}},"frontmatter":{"title":"Sooftware Speech - Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition Paper Review","userDate":"17 March 2021","date":"2021-03-17T10:00:00.000Z","tags":["speech","paper"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/7ca66667228d877c1b0d96e85b974435/ee3dd/pushing.png","srcSet":"/static/7ca66667228d877c1b0d96e85b974435/6a16f/pushing.png 750w,\n/static/7ca66667228d877c1b0d96e85b974435/ee3dd/pushing.png 928w","sizes":"100vw"},"sources":[{"srcSet":"/static/7ca66667228d877c1b0d96e85b974435/e0c95/pushing.webp 750w,\n/static/7ca66667228d877c1b0d96e85b974435/86029/pushing.webp 928w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6142241379310345}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":20,"edges":[{"node":{"id":"fa9e8cbb-841a-516f-9df6-4be0336b56b0","excerpt":"한국어 Tacotron2 이번 포스팅에서는 Tacotron2 아키텍처로 한국어 TTS 시스템을 만드는 방법에 대해 다루겠습니다. Tacotron2 Tacotron2는 17년 12월 구글이 NATURAL TTS SYNTHESIS BY…","frontmatter":{"title":"Sooftware Speech - 한국어 Tacotron2","date":"2021-10-10T10:00:00.000Z"},"fields":{"readingTime":{"text":"11 min read"},"slug":"/korean_tacotron2/"}}},{"node":{"id":"43c23529-71b1-5d60-8883-a45fbcd55ebd","excerpt":"Textless NLP: Generating expressive speech from raw audio paper / code / pre-train model / blog Name: Generative Spoken Language Model (GSLM…","frontmatter":{"title":"Sooftware NLP - Textless NLP","date":"2021-09-19T10:00:00.000Z"},"fields":{"readingTime":{"text":"4 min read"},"slug":"/Textledd NLP: Generating expressive speech from raw audio/"}}},{"node":{"id":"83c6b4fa-d71e-51d8-90bb-b58bfffc01d0","excerpt":"Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition Yu Zhang et al., 2020 Google Research, Brain Team Reference…","frontmatter":{"title":"Sooftware Speech - Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition Paper Review","date":"2021-03-17T10:00:00.000Z"},"fields":{"readingTime":{"text":"3 min read"},"slug":"/Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition/"}}},{"node":{"id":"894a24ca-fff9-5884-b6da-1c97f2ece7bc","excerpt":"PORORO Text-To-Speech (TTS) 얼마전에 저희 팀에서 공개한 PORORO: Platform Of neuRal mOdels for natuRal language prOcessing 라이브러리에 제가 공들여만든 TTS…","frontmatter":{"title":"PORORO Text-To-Speech (TTS)","date":"2021-02-16T10:00:00.000Z"},"fields":{"readingTime":{"text":"1 min read"},"slug":"/pororo-tts/"}}},{"node":{"id":"b039977c-cecb-50f4-a0dd-fd008731bc99","excerpt":"EMNLP Paper Review: Speech Adaptive Feature Selection for End-to-End Speech Translation (Biao Zhang et al) Incremental Text-to-Speech…","frontmatter":{"title":"Sooftware Speech - EMNLP Paper Review: Speech","date":"2020-12-08T10:00:00.000Z"},"fields":{"readingTime":{"text":"4 min read"},"slug":"/2020 EMNLP Speech Paper Review/"}}}]}},"pageContext":{"slug":"/Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition/","prev":{"excerpt":"카카오브레인 퇴사, 그리고 창업 (feat…","frontmatter":{"title":"카카오브레인 퇴사, 그리고 창업 (feat. 졸업)","tags":["record"],"date":"2021-02-27T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsTAAALEwEAmpwYAAADRUlEQVQozxXQW09bBQDA8fPoux/CxGBmonuYwckKZZHJg0aTBTXRTBazRKcs2ySZM9x2AToGbNO5rAwXGCC6wihQLJQWaOmd9hx6OaXQjrYbpaWl5W7H3/j7CD8heEdJf62Sovcq6aypYHXgAhvjHSQt9xBvK/G1lyK1lyKqPsBd+ymOK9X4W0pwd3/LukaF/1YZtmvv4mgpwdV6HMF68wRyZzEPasoZu9/Enk2HZ0FmLZPg+dNLWFQVqGveR2yvwHyxCt25Lwl3fsysyUp6cgTraB9u9VmkDiWxrm8QnC3vkO1/jdRgEenuJrL9D1mMJ0nm90jINgYftRGcH2Z/1ciONMS2vYeMt5cVrwXJ6WPQWcBsWmD3WRux3l8QUsNvw4TAqxGB/Hg52wkvqViQ9YhIISlCxgdbIQoxkcN8jJzVTCHqhKzIVtyLHAwgupewGx3Mt51CsDytozBzFLQC+1Nvsh8zsBfX82rNgXduiLjPBJteYrNdZBZcsCETmB9hxTMJ2QAHkWmmH//EbNd5XNePIjhVSjz3ykn1vc52aJjDtI942EBu1Y7D8Bd6jRqj9g9WxXESkoGNZTNxv4nMsgUyMlvyP0xcU9DS8Am65pMIYdVnSFcVuH89TSEXgawft0XLkmeKXMSK3zONbB/DM6Ph2cDvhBeMkJchLUEmQMrVR/+FIww1KbCqTiCMnvsa0+Xvya042E1u8m8yRD40Q0YykI87eaA3MWyeIxmaQ3bPwuJl7usa2XgZgoMXyDo1M42fY+v4Am3dSQRbTQnLQ71E84c816jZjy0yF3HS5zFA1EJUNPJjzxhGy/9nIXZtjUw8uc7m+jLZ7R2ingnSQw00/HCGomMVCLN1bxGZukPuAJbiUbaCet6IubjkN4J1EOx/kkpI7K4tsvvCQz4RBGmSnZm/8YdjrCYjpObVWJpPob1yDMH1WxVmOYA7MI/dH8D1pIWLd+txaRtIaOsRb11lw/KQTXGAnH+EnaiRLZ8OJh+RjMq83Ialsbs4bxaz2F6K4O8oY6BHRbfeiPT4PGPVp/H+/BHe9lLsrQpGz1YxXVuJQ3UcW6sSe2sZ+huVnKn+juHWrwho6nHc/hBHczEulYL/AGz28AsxF5MEAAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/326851fe970bf69bd6fc9b3f450fe424/776e5/new_start.png","srcSet":"/static/326851fe970bf69bd6fc9b3f450fe424/7f1c2/new_start.png 750w,\n/static/326851fe970bf69bd6fc9b3f450fe424/776e5/new_start.png 960w","sizes":"100vw"},"sources":[{"srcSet":"/static/326851fe970bf69bd6fc9b3f450fe424/754fd/new_start.webp 750w,\n/static/326851fe970bf69bd6fc9b3f450fe424/b8767/new_start.webp 960w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5916666666666667}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"4 min read"},"layout":"","slug":"/new_start/"}},"next":{"excerpt":"GPT Understands, Too Xiao Liu et al. Tsinghua University etc. arXiv pre-print Abstract GPT를 파인튜닝하는 방법은 Narural Language Understanding (NLU…","frontmatter":{"title":"Sooftware NLP - P-Tuning Paper Review","tags":["nlp","paper"],"date":"2021-05-13T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAAAsTAAALEwEAmpwYAAABE0lEQVQY002P226DMBBE+f9vqvrYm9SKpGml5gqkIeBgF2JjE2xyKmgrdaV9mNmz0kwUwsAwDDjXoY2h7y+E/9p7TNuim5p6P8fJDB+GyW+rhCaLcabhrA3X60DkbEutJOXxwDHPyYsTzlmqsuRUllyMYZwRdtZgdENxzFlvU7TWBH+hEoJstyOEQNSIjOLjAbG857B55ealJFm/U8WPVIsFfVGgz5qmVqh0hljeka3m3M4qktUbaveEmD0j4pgQPFHnHEPw2NaQpSm7JEUphe97xlvr3JSwH3X3wyopWW+2fO73NM0XQwi0xtB1HdEYc4SllORj5fyAlGryrLV4738rX+l7P+lxhSgRQvD331o7Vf4GH4d759sk/s8AAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/5251bff33539be461c690325c488ea29/c0e65/p_tuning.png","srcSet":"/static/5251bff33539be461c690325c488ea29/f0b2b/p_tuning.png 750w,\n/static/5251bff33539be461c690325c488ea29/3831f/p_tuning.png 1080w,\n/static/5251bff33539be461c690325c488ea29/ec348/p_tuning.png 1366w,\n/static/5251bff33539be461c690325c488ea29/c0e65/p_tuning.png 1920w","sizes":"100vw"},"sources":[{"srcSet":"/static/5251bff33539be461c690325c488ea29/a4cf2/p_tuning.webp 750w,\n/static/5251bff33539be461c690325c488ea29/4e6df/p_tuning.webp 1080w,\n/static/5251bff33539be461c690325c488ea29/04ab9/p_tuning.webp 1366w,\n/static/5251bff33539be461c690325c488ea29/19b4f/p_tuning.webp 1920w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.23489583333333336}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"4 min read"},"layout":"","slug":"/p_tuning/"}},"primaryTag":"speech"}},
    "staticQueryHashes": ["3170763342","3229353822"]}